Representation Learning by Detecting Incorrect Location Embeddings

In this paper, we introduce a novel self-supervised learning (SSL) loss for image representation learning. There is growing belief that generalization in deep neural networks linked to their ability discriminate object shapes. Since shape related the location of its parts, propose detect those have been artificially misplaced. We represent parts with tokens and train ViT which token has combined an incorrect positional embedding. then sparsity inputs make model more robust occlusions speed up training. call our method DILEMMA, stands Detection Incorrect Location EMbeddings MAsked inputs. apply DILEMMA MoCoV3, DINO SimCLR show improvement performance respectively 4.41%, 3.97%, 0.5% under same training time linear probing transfer on ImageNet-1K. also full fine-tuning improvements MAE ImageNet-100. evaluate via common SSL benchmarks. Moreover, when downstream tasks are strongly reliant (such as YOGA-82 pose dataset), pre-trained features yield significant gain over prior work.